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ABSTRACT 



A mathematical programming approach is presented for 
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attributes. Because mathematical programming problems have to be solved while 
the examinee waits for the next item, a fast implementation of the 
Branch-and-Bound algorithm is needed for this approach. Eight modifications 
of the algorithm especially designed for application in CAT are described. In 
order to investigate the effects of the modifications, two empirical examples 
using simulation were studied. The modified Branch-and-Bound algorithm 
selected the items in the adaptive tests in a realistic amount of time, while 
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Abstract 

A mathematical programming approach for computer adaptive testing (CAT) with 
many constraints on the item and test attributes is presented. Because in the approach, 
mathematical programming problems have to be solved while the examinee waits for 
the next item, a fast implementation of the Branch-and-Bound algorithm is needed. Eight 
modifications of the algorithm specifically designed for application in CAT are described. 
In order to investigate the effects of the modifications, two empirical examples were 
studied. The modified Branch-and-Bound algorithm selected the items in the adaptive 
tests in a realistic amount of time, while the resulting tests met the constraints. 
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Introduction 

In Computer Adaptive Testing (CAT), a number of approaches have been proposed 
to deal with content constraints on adaptive tests. Both Kingsbury and Zara (1991) 
and Segall, Moreno, and Hetter (1997) suggest to partition the item pool and to use an 
algorithm to select items from these pools during the CAT or even to use separate adaptive 
tests and to combine final scores afterwards. In Stocking and Swanson (1993) the items 
are selected sequentially, minimizing the weighted sum of deviations from the constraints. 
Recently, a new approach to constrained CAT based on the assembly of shadow tests prior 
to item selection, was introduced in van der Linden and Reese (1998). 

The following pseudo algorithm for this Shadow Test Approach is given in van der 
Linden (2000): 

Step 1: Initialize the ability estimator. 

Step 2: Assemble a shadow test that meets the constraints and has maximum 
information at the current ability estimate. 

Step 3: Administer the item in the shadow test with maximum information at the 
ability estimate. 

Step 4: Update the ability estimate. 

Step 5: Return all unused items to the pool. 

Step 6: Adjust the constraints to allow for the attributes of the item administered. 

Step 7: Repeat Steps 2-6 until m items have been administered. 

If the item pool is well designed, the shadow test approach guarantees that all 
constraints are met for every CAT assembled. In order to gain these profits an integer 
programming formulation of the test assembly problem in Step 2 is necessary. 

Several algorithms have been proposed for solving integer programming problems 
in test assembly, for example, those in Contest (Timminga, van der Linden & Schweizer, 
1996), the heuristic for the Weigthed Deviation Model (Swanson & Stocking, 1993) and 
several implementations of the Network-Flow algorithm by Armstrong, Jones & Whng 
(1995). These algorithms have proved to be very useful in test assembly, but they may 
result in occasional constraint violation, or in a sub-optimal solution. So, when they 
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are applied it is not always possible to gain full profits of the shadow test approach. 
Alternatively, a Branch-and-Bound (B&B) algorithm can be used to find the optimal 
solution (Adema, 1992, Timminga, van der Linden, and Schweizer, 1996). However, 
the B&B algorithm is rather time consuming. Especially for laige adaptive tests or tests 
with an item-set structure, this might be a problem since in CAT only a small amount of 
time is available for selecting the next item. 

The goal of the present research is to find modifications of the B&B algorithm such 
that the shadow test approach can be used for the problem of assembling adaptive tests 
with large number of constraints from large item pools. In this paper, first a general 
formulation of a mathematical programming model for selecting the k - th item in an 
adaptive test as well as the B&B algorithm is given. Then, eight modifications of the 
algorithm applicable in a CAT-framework are described. To investigate the effect of these 
modifications, two test assembly problems were studied, and their results are presented. 
Finally, the modifications are evaluated and some recommendations about their use are 
given. 



Mathematical Programming Model 

In order to apply mathematical programming techniques for solving the problem of 
selecting the next item, the problem has to be written in a specific format. An objective 
function that will be optimized, has to be defined and the test specifications have to be 
written as constraints. Let, 

h{0k- i) be the information provided by item i at the estimated ability level 9 k- 1 , 

N be the number of items in the item pool, 
i = 1, ..., N be the index ranging over all items, 

S be the number of sets in the pool, 
s = 1, ..., S be the index ranging over all item sets, 

Xi and y s denote 0-1 decision variables for selecting item i or item set s, 

n mm minimum number of items in set s to be selected when s is in the test, 

n max jjg max i mum number of items in set s to be selected when s is in the test, 
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V g be a subset of indices of sets with attribute g, 

Vj be a subset of indices of items with attribute j, 

V a be a subset of indices of items in set s, 
k be the index ranging over the items in the adaptive test, 
Sk - 1 be the set of indices of k — 1 item already selected, 
a s , b{ be values of the set attributes and the item attributes, 
n g , rij be attribute values of the entire test, 

Ti be the fixed total test length. 



Model 

The model for selecting the k-th item in the adaptive test can be formulated as 
follows: 
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B&B Algorithm 

It is important to notice that both variables Xi and y s are 0-1 variables. So, the B&B 
algorithm can be applied. Intuitively, the algorithm can be described in the following 
manner. The B&B algorithm starts with the computation of the optimal solution to the 
relaxation, i.e., the problem where the integer constraints in Equation 2f are left out. The 
problem is now in the class of linear programming problems, which are much easier 
to solve. The computation is done by the simplex method, a well known method for 
solving linear programming problems (Winston, 1994). Since the set of constraints of 
the relaxation is smaller than and contained within the set of the original problem, the 
objective value of the relaxation will be higher, and it can be used as an upper bound z for 
the solution of the test assembly problem. When the solution of the relaxation only has 
integer values, the optimal solution to the test assembly problem is already obtained. If 
not, the problem is branched in two new problems. In one problem a non-integer variable 
is fixed at zero. In the other this variable is fixed at one. These new problems are treated 
the same way, until all branches are searched. Whenever a solution is infeasible, i.e., 
conflicts with the constraints, or when the value of the relaxation (z) is smaller than the 
lower bound ( z ), i.e., the best integer solution found so far, this branch is pruned. The 
reason is that it is no longer possible to meet the constraints or to find a better solution, 
because some variables Xi have been fixed. The remaining problems are branched by 
fixing new variables, and bounds for the new problems are calculated. When a better 
integer solution is found, the value of z is updated. The B&B method terminates when 
all branches have been investigated or pruned. 

Example 

To illustrate the B&B algorithm, a small test assembly problem is introduced. The 
objective function in this problem is to optimize Fisher’s information function at a certain 
ability estimate. Three constraints are involved. Item 1 and 2 are in the same content 
class. From this content class at least one item should be in the test. The word count for 
the total test should be less or equal to 150. Finally, since Item 2 and Item 3 contain clues 
to each other, only one of them can be in the test. The 0-1 LP formulation of this problem 
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is given in Equation 3 through Equation 7: 

max0.805xi + 1.158x2 + 0 . 753 x 3 (3) 

subject to: 



Xi + x 2 
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1, 
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1 , 


( 6 ) 
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The B&B search is presented in Figure 1 . The upper half of a node consist of a 
problem number and the objective value of the relaxation; the lower half contains the 
solution of the relaxed problem. The optimal solution is in Node 4, which has a feasible 
solution with the highest value for the objective function. 



Insert Figure 1 at about here 



Pseudo Algorithm 

The B&B method can be described by the following pseudo algorithm. 

Step 1. Select a node and solve the relaxation associated with it. 

Step 2. When the value of the relaxation is smaller than the lower bound for the 
optimal solution, no better solution can be found in this branch. Prune the branch. Go to 
Step 6 . 

Step 3. When the value of the relaxation is greater than the lower bound, but the 
values of x, are not integer, the branch might contain an improvement. Go to Step 5. 

Step 4. When the solution of the relaxation is feasible and is better than the best 
solution found so far, adjust the lower bound. Delete all nodes with upper bounds lower 
than the new lower bound. Go to step 6 . 
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Step 5. Branch the node. In one branch the value of Xi is set equal to x< = 0. In the 
other branch x< = 1. 

Step 6. Stop when all branches are investigated or pruned, otherwise go to Step 1. 

A formal description of the algorithm is given in Nemhauser & Wolsey (1988, page 
355). 

In the next section eight modifications of this general algorithm are described. 

Modifications of the B&B algorithm 



The B&B algorithm is in the class of implicit enumeration methods. Algorithms in 
this class search the set of possible integer solutions in such a way that not all individual 
possibilities need be considered. Nevertheless their worst-case performances sometimes 
are considerably time consuming. This feature of the B&B algorithm is especially 
unfavorable for CAT, where the time available for selecting the next item is limited. In 
this context, accuracy and speed are essential to apply an algorithm. Therefore, measures 
to improve the performance of the B&B algorithm were investigated. From the theory of 
integer programming several improvements were derived. Eight of such improvements 
are described below. 

1. Starting Solution 

The first modification is the use of a starting solution. In the general algorithm no 
starting solution is used and the lower bound is set equal to z = — oo. Whenever an integer 
solution to the problem is known, it can be used as a starting solution and this solution 
will provide a better initial lower bound. 

This observation can be used for speeding up the B&B algorithm. As already noted in 
Step 2, a branch is of no interest and can be pruned, when the upper bound of the objective 
values of solutions in the subset is smaller than a lower bound to the objective value of 
the optimal solution. Using a good starting solution might provide a strong lower bound 
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and hence a lot of branches might be pruned. So fewer branches have to be investigated 
and the algorithm finishes earlier (see also Adema, 1992). 

Especially for CAT, this feature might be of interest. When the shadow test approach 
(van der Linden and Reese, 1998) is used and the constraints do not depend on the 
estimated theta 6, the solution to the problem in Iteration k — 1 can be used as a 
starting solution to the problem in Iteration k, because the constraints in both problems 
are generally identical. After a number of items, the ability estimate of the examinee 
stabilizes and the starting solution even provides stronger lower bounds. First remind 
that the problem in Iteration k is based on the ability estimate 6 k- 1 , and the problem in 
Iteration k — 1 is based on the ability estimate 6k- 2 . When the ability estimate stabilizes 
after a number of items, 6k-i is not much different from 6k- 1 . Therefore, the objective 
function in Iteration — 1 is not much different from the objective function in Iteration k, 
and since the constraints in both iterations are generally equal, the solution to the problem 
in Step k — 1 provides a good approximation to the solution of the problem in step k. 

When no prior information about the examinee is known, the initial estimate is often 
set equal to 6 q = 0 for all examinees. For this initial estimate, the first item in a CAT 
can be calculated in advance. Once available, it can be used as a starting solution for all 
examinees. When a variable initial ability estimate is used, a different strategy should 
be followed. Instead of calculating one starting solution in advance, starting solutions 
should be calculated for a number of points on the ability scale 6oj, j = \..J, that cover 
a sufficiently large part of the ability range. The solution for doj closest to the initial 
estimate of the examinee, 60, can be used as a starting solution. 

2. Integer slack variables 

A second improvement (Williams, 1990, page 204) deals with formulation of the 
constraints. Consider an arbitrary constraint ^ a^Xi < b, where a* s and b are integer 
constants and Xi € {0, 1}. An example is the word count constraint given in Equation 5, 
where a* represent the number of words per item, and b is the maximum number of words 
in the test. Other examples are time-limit constraints, content constraints, constraints on 
gender-orientation of the items, or on item format. 
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Because the variables Xi are integer, the difference b — will also be integer. 

So, it is possible to introduce an integer slack variable 6, and reformulate the constraint 
as 



7: a,iXi + 6 = b. (8) 

During the B&B search, one could also branch on these integer slack variables. 
When applied to the problem in Equation 1 through Equation 2f, two versions of this 
modification are needed: 



^ tisUs + &q — n g 


(9) 
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E bi Xi + Sj = rij 


(10) 


ieVj 





Both at set level and at item level some constraints can be rewritten as equalities. 
There is advantage in introducing integer slack variables and giving them priority in the 
branching process (see next modification). This idea is due to Mitra (1973). 

3. Priority ordering 

A general recommendation concerning B&B methods is that one should branch 
on variables with the greatest impact first (Williams, 1990, page 205). Applying this 
recommendation to the general algorithm involves a more detailed specification of Step 
5. In Step 5 the set of possible solutions is split in two parts by fixing a variable. The 
algorithm should be modified such that a node is divided into two branches by fixing a 
variable with high priority. 

The gain involved in this modification can be illustrated by considering an assembly 
problem where both set and item variables are present. In this case, difference between 
the impact of their variables exists. Fixing item variables has little effect, but fixing set 
variables forces the algorithm to select items from the sets that are chosen to be in the 
test. Also, if set variables are fixed at zero, all item variables that belong to these sets 
are automatically fixed at zero. A different example deals with the integer slack variables 
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introduced above. Integer slack variables 6 often have greater impact on the problem than 
the other variables in the constraint. In a content constraint, the slack variable determines 
the number of items or number of sets with a certain content to be in a test. 

4. Branching strategy 

Now that a priority order can be made, Step 5 of the algorithm could be specified 
more in detail. Variables with high priority should be chosen first. However, the question 
remains which variable to choose in the same priority class. In modem IP-solvers some 
options are provided (BLOG, 1999). The solution of the LP-relaxation xrp, calculated 
in Step 1, is often used as an indicator. In this relaxation, the variables do not have to 
take integer values, so the variables x' RP can be written as the sum of an integer and a 
fractional part, x^p = ipp + /jjp, where 0 < /pp < 1. 

The Maximum Infeasibility Rule. This rule branches on the variable whose 
fractional value f l RP deviates most from zero and one. In this way, all variables get integer 
values slowly but surely. 

The Minimum Infeasibility Rule. This rule branches on the variable whose fractional 
value f RP deviates less from zero or one. The idea is to find a good integer solution to 
the problem in a small amount of time. This solution provides a strong lower bound and 
branches are pruned earlier. 

More Intelligent Branching Rules. Rules, like branching on pseudo costs or on 
pseudo reduced costs (Adema, 1992, Forrest, Hirst and Tomlin, 1974, Mitra, 1973), can 
also be applied. These rules generally need more time to find a sharp lower bound, but 
often perform better over all. 

Automatic Selection of Branching Variable. This rule is a combination of the above 
described rules. An algorithm determines which rule seems most promising and decides 
which rule is applied. 

In CAT, a choice should be made between the intelligent branching rules and the 
minimum infeasibility rule. The maximum infeasibility rule will probably take to much 
time. If time is very limited the minimum infeasibility rule will probably perform better, 
because its first shot often provides a good solution. If a little more time is available the 
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intelligent branching rule is more likely to produce the best result. Its first shot may be 
worse than for the minimum infeasibility rule but it will find improvements faster and 
eventually result in a better solution. 

5. Node Selection 

In Step 1 of the general algorithm, a node is selected to be investigated. Instead 
of randomly choosing a node, more advanced strategies can be used. The depth-first 
strategy, the best-bound strategy and the best-estimate strategy (Nemhauser and Wolsey, 
1988, pages 358-359, e.g.) are reasonable alternatives. 

The Depth-First Strategy. This strategy selects the most recent node. Whenever this 
strategy is used, a different variable is fixed in every iteration. In this way, an integer 
solution is found in a small amount of time. Unfortunately, in combination with the use 
of a priority ordering, this strategy has a serious drawback. Due to the priority ordering, 
variables with high priority are fixed first and variables with lower priorities last. After 
finding the first integer solution, the most recent problems result from fixing a low- 
priority variable. In the subsequent iterations, only the values of the variables with low 
priority are changed. So both modifications counteract. 

The Best-Bound Strategy. This strategy selects the node with best optimal value for 
the relaxed problem. Whenever the relaxed problem provides a good upper bound, the 
difference between this upper bound and the lower bound is a good indication of the 
improvement that can be reached by selecting this node. 

The (Alternative) Best-Estimate Strategy. This strategy selects a node by estimating 
its best integer solution. This estimate is made by correcting for the fractional values of 
the variables x (e.g. see Mitra, 1973, Nemhauser & Wolsey, 1988). 

In a CAT program with item sets, some variables have higher priorities than others. 
Therefore, the depth-first strategy is unsuitable for obtaining results in a small amount of 
time. Both remaining strategies have their pro’s and con’s. The best-bound strategy has a 
time advantage, because the objective values of the relaxations are known. On the other 
hand, the relaxed problem does not provide a very good bound. The best-estimate strategy 
performs better in selecting promising problems, but more time is needed for calculating 
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the estimates. So, no theoretical arguments can be offered to prefer one strategy over the 
other. 

6. Special Ordered Sets (SOS) 

The next modification deals with a special kind of constraints. A special ordered set 
of Type 1 (Beale and Tomlin, 1969) is a set of variables from which at most one can be 
equal to one: 



When these kind of constraints are in a problem, the fifth step of the algorithm can be 
made more efficiently (e.g., Forrest, Hirst and Tomlin, 1974). In Step 5 of the algorithm 
a node is branched by fixing a variable at X* = 0 or at x* = 1. However, instead of 
branching a node by fixing a single variable, it is possible to fix half the number of items 
in the special ordered set. First split the variables XjJ = 1 , k into two subsets x], 
where j = 1, ..., / and x ] .where j = / + 1, ..., k. Since only one variable in the entire set 
can be equal to Xj = 1, this variable is either in the first half or in the second half of this 
set. So, either 
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(ii) 



o j = l,...,/, 
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Xj < 1. 



j=f + 1 



or 



/ 



^ ' Xj < 1, 



0 j = f + l,...,k. 



In this way the node can be branched, and half the number of variables, rather than one, 
is fixed in the same iteration. 



